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Abstract 

The goal of the two Power of Data 
(POD) projects was to increase science, 
technology and math skills through the 
implementation of project-based learning 
modules that teach students how to solve 
problems through data collection and 
analysis utilizing geospatial technologies. 
Professional development institutes in two 
formats were offered to encourage teachers 
to implement the modules. We compared 
teacher learning, teacher implementation, 
and student learning from the two different 
professional development formats to ex¬ 
amine how each format supported teachers 
to implement the modules, and, ultimately, 
improve student understanding. Teacher 
surveys, content and technology assess¬ 
ments, classroom observations, student 
assessments, and student work samples 
were analyzed for comparison between a 
two-week summer institute and monthly 
meetings held throughout the academic 
year. Teachers and students from both 
formats showed improvement in all areas 
assessed, yet there was not a large effect 
on student outcomes based on the delivery 
format between the professional develop¬ 
ment sessions. 

Introduction 

A common goal of professional de¬ 
velopment (PD) is to improve student 
learning, but there are many mediating 
factors between teacher experiences 
during the PD, levels of implementa¬ 
tion, and subsequent student learning in 
the classroom. Technology further com¬ 
plicates implementation due to factors 
such as teachers’ content knowledge, 
support at the school site, and class¬ 
room resources (Mumtaz, 2000; Tamim, 
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Bernard, Borokhovski, Abrami, & 
Schmid, 2011). The purpose of this 
study is to examine how two different PD 
formats support teachers to implement 
technology into classrooms to improve 
student understanding. The assumption 
is that changes in student understanding 
are only observed after teachers imple¬ 
ment the professional development in¬ 
tervention, so if student understanding is 
to increase, we must first improve teach¬ 
ers’ understanding and skills as needed 
to change classroom practices. 

The two Power of Data (POD) proj¬ 
ects sought to increase science, technol¬ 
ogy, and math skills through learning 
modules that teach students how to 
solve problems through data collection 
and analysis utilizing geospatial tech¬ 
nologies. The PD team included geology 
faculty, science teacher professional de¬ 
velopers, geospatial technology experts, 
evaluators and science education re¬ 
searchers. The focus of the POD PD was 
to improve teacher pedagogy through 
the use of ArcGIS software technology, 
while purposefully alleviating barriers 
to implementation of new technology. 
During the PD institutes, time was spent 
emphasizing student projects related to 
claims and evidence, data collection, 
and presentation using geospatial tech¬ 
nology. With equivalent information 
presented and resources afforded to each 
set of teachers during PD, the question 
remained: Did the delivery format of 
the teacher PD institute affect teacher 
and student program outcomes? To an¬ 
swer this question we compared teacher 
learning, teacher implementation, and 
student learning from two different PD 
formats. One form of the PD was pro¬ 
vided through an intensive two-week 
summer institute (SI), and the other uti¬ 
lized monthly or bimonthly meetings 
throughout the academic year (AY). 


POD Professional Development 
Model 

The POD PD model was based on re¬ 
search on best practices in professional 
development for teachers in which tech¬ 
nology is integrated into the classroom 
instruction (Flick & Bell, 2000; Kerski, 
2003; Loucks-Horsley, Love, Stiles, 
Mundry, & Hewson, 2003; National 
Research Council [NRC], 2007; Parker, 
Carlson, & Na’im, 2007; Ringstaff & 
Kelley, 2002). Research on models of 
teacher change includes the follow¬ 
ing variables to account for levels of 
implementation: teachers’ beliefs and 
attitudes (Lawless & Pellegrino, 2007; 
Mumtaz, 2000), fears of failure (Lawless 
& Pellegrino, 2007; Mumtaz, 2000), 
comfort level (Barnett & Mark, 2010; 
Lawless & Pellegrino, 2007; Kubitskey, 
Fishman, & Marx, 2002; Trautmann & 
MaKinster, 2009), and pedagogical con¬ 
tent knowledge (Gess-Newsome, 2002; 
NRC, 2007). Barriers to implementation 
are further increased when technology 
is involved (Cuban, Kirkpatrick, & Peck, 
2001; Lawless & Pellegrino, 2007). 
Additionally, research suggests that the 
capacity of technology to improve stu¬ 
dent learning depends more on teacher 
pedagogy, content knowledge, and in¬ 
structional goals than the design of the 
technology itself (Tamim, Bernard, 
Borokhovski, Abrami, & Schmid, 2011). 
Therefore, the POD PD was developed 
to support teachers to implement ef¬ 
fective pedagogy, alleviate barriers to 
implementation, and use the technology 
in the classroom based on the research. 
Based on the literature review, the fol¬ 
lowing consolidated categories were 
examined: 

1. teacher learning of content, peda¬ 
gogy, technology, and teacher 
satisfaction following the profes¬ 
sional development; 
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2. teacher implementation of content 
and pedagogy presented in the 
projects; 

3. barriers and pathways of this 
implementation; and 

4. student learning. 

These categories both informed the 
design of the POD PD model and the data 
that were gathered to determine the ef¬ 
fectiveness of the PD. Each category will 
be discussed in terms of the research on 
the topic and how it was addressed in the 
POD PD design below. 

Teacher Learning 

Features of PD that have positive effects 
on teachers’ self-reported knowledge and 
skills and changes in classroom practice 
include: active learning, opportunities 
to collaborate with colleagues, time for 
metacognitive activities, and alignment 
with curricula and other professional 
development experiences (Garet, Porter, 
Desimone, Birman, & Yoon, 2001; Baylor 
& Ritchie, 2001; Mumtaz, 2000). Within 
projects that integrate technology, the 
greatest influence on teacher practice 
occurs when teachers participate in pro¬ 
fessional learning experiences in which 
they are: 1) provided technology-enhanced 
teaching materials; 2) required to imple¬ 
ment with some freedom to customize 
the materials to their individual context; 
and 3) provided time with colleagues 
to reflect upon the experience. The PD 
must also provide support so that teach¬ 
ers are able to use evidence of student 
learning to improve their teaching prac¬ 
tices (Gerard, Varma, Corliss, & Linn, 
2011; Trautmann & MaKinster, 2009). 
Additionally, one of the critical factors 
that determines the effectiveness of a 
professional learning experience is par¬ 
ticipant satisfaction (Guskey, 2000). As 
teachers have an opportunity to build 
their knowledge of content and skills 
with technology, their morale increases 
(Baylor & Ritchie, 2002; Garet et al., 
2001). Therefore, the focus of the POD 
PD design was on teacher learning and 
satisfaction with the PD to lead to imple¬ 
mentation in the classroom. 

The PD was designed in a manner to 
scaffold teacher learning. All teachers 
who attended received twelve books on 


the topics of: teaching and learning with 
geographic information systems (GIS) 
software, effective science and math 
pedagogy, formative assessment strate¬ 
gies, and research on teaching and learn¬ 
ing. The teachers also received 1) six 
sets of Vernier LabQuest data collection 
devices with GPS and soil moisture and 
temperature measurement capabilities 
and 2) site licenses for ArcGIS with ex¬ 
tensions and Logger Pro software. 

During the PD, teachers experienced 
lessons as learners first, before imple¬ 
menting the lessons in their classrooms. 
The goal was that they would know what 
to expect in terms of struggles their stu¬ 
dents might experience, while increasing 
their own confidence, content knowledge, 
and technical skills. Teachers received 
lessons to implement that could be taught 
immediately, or that could be modified 
to better meet their instructional needs. 
Participants were given time to plan how 
they would implement these lessons, and 
were provided an online space where they 
could discuss implementation, ask ques¬ 
tions of each other about difficulties, and 
share successes. Following the institutes, 
monthly discussion board questions were 
posed to the teachers to encourage col¬ 
laboration amongst participants. 

Implementation 

Implementation is considered the key 
step between teacher PD and student 
learning. Factors that affect implemen¬ 
tation include teachers’ content knowl¬ 
edge, support at the school site, and 
classroom resources. When implement¬ 
ing technology-enhanced lessons, the 
first year is typically spent overcoming 
barriers, such as issues with hardware 
and software, suggesting that programs 
of duration of more than one year are 
most effective in changing teacher prac¬ 
tice (Gerard et al., 2011). Implementa¬ 
tion and integration of technology is 
also influenced by a teacher’s openness 
to change (Baylor & Ritchie, 2002). 
Activities that focus on specific content, 
are tied to relevant school reform efforts, 
promote student-centered teaching, and 
are aligned to curricula are more likely 
to be viewed positively by participants 
as more applicable to their classrooms 


(Baylor & Ritchie, 2002; Desimone, 
Porter, Garet, Yoon, & Birman, 2002; 
Garet et al., 2001; Guskey, 2002). Strong 
educational technology leaders, such 
as those that value and utilize technol¬ 
ogy and include the use of educational 
technology in school plans, can increase 
teachers’ perceptions of student content 
acquisition (Baylor & Ritchie, 2002). To 
address these factors, both PD formats 
were designed to alleviate fears, reduce 
costs, provide teachers with planning 
time, and create support at the school 
site. PD recruitment efforts focused on 
two areas: 1) support at the school site; 
and 2) the teacher’s openness to and ex¬ 
perience in teaching with technology in 
the classroom. 

Because technology was added on top 
of the already high demands of new peda¬ 
gogies involved in student-centered and 
project-based instruction, barriers to im¬ 
plementation, whether perceived or actual, 
were anticipated. To mitigate these effects, 
several supports were in place before, dur¬ 
ing, and after the PD institutes. To ensure 
support at the school level, pairs of teachers 
from the same school were invited to ap¬ 
ply, with the preference of one Career and 
Technical Education (CTE) teacher and 
one science or math content teacher. The 
idea was that the CTE teacher might have 
more confidence with technology and/or 
project-based instruction, and the content 
area teacher more confidence with the sci¬ 
ence or math subject matter. Teachers were 
asked to confirm basic technology skills 
such as navigating to files, saving files, and 
downloading data from the Internet. They 
were also asked to provide examples of 
their teaching that included inquiry-based 
lessons, technology and subject integra¬ 
tion, and collaborations with other teach¬ 
ers. Those who had more experience or 
who had a partner at their school were se¬ 
lected over those with less experience with 
technology and inquiry-based pedagogy or 
with no partner. 

All accepted participants confirmed they 
had access to a Windows-based computer 
lab necessary to run ArcGIS software and 
other basic system requirements. Ad¬ 
ditionally, all participants were required 
to provide a signed memorandum of 
understanding from their school or district 
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information technology (IT) department 
representative. The memorandum was 
crafted by our technology lead and de¬ 
scribed specific recommendations for 
the lab. It included details on storing and 
maintaining data, loading software, and 
deadlines by which the lab would need to 
be ready to use. Additionally, PD and tech¬ 
nical staff were available to answer ques¬ 
tions and assist with software installation 
and troubleshooting. 

Study Overview 

A review of research on the effective¬ 
ness of technology-integrated and sci¬ 
ence, technology, engineering, and math 
(STEM) PD tends to support the notion 
that context and content within the pro¬ 
fessional learning experiences are more 
important than format of the PD activi¬ 
ties. However, the majority of these stud¬ 
ies are based on teacher self-report rather 
than student outcomes (Garet et al., 2001; 
Guskey, 2002; Guskey, 2003; Kennedy, 
1998; Mumtaz, 2000). Yet, self-report is 
not enough if we want to improve student 
learning in STEM classes following PD. 
Our focus then was to examine the effect 
that technology-integrated, project-based 
learning modules have on the learning of 
secondary students following PD. 

The study design was based on Guskey’s 
(2000) evaluation of teacher PD, which 
describes “five critical levels of informa¬ 
tion.” In Guskey’s model, the emphasis 
is on teacher learning, implementation, 
and student learning. The five levels for 
PD evaluation are: 

1. participants’satisfaction, 

2. participants’ learning, 

3. organizational support and change, 

4. participants’ use of new knowl¬ 
edge and skills, and 

5. student learning outcomes. 

Each level increases in complexity and in¬ 
fluences the success of each higher level. 
In other words, the first four levels affect 
the fifth level. The premise is that effec¬ 
tive PD (Levels 1 and 2) coupled with 
administrative support (Level 3) can influ¬ 
ence teachers’ classroom practices (Level 
4) which then change student learning 
outcomes (Level 5) thereby influencing 
changes in teachers’ beliefs and attitudes 
(Guskey, 2002). 


The original intent of both POD projects 
was to provide on-going professional learn¬ 
ing experiences throughout the academic 
year (AY) based on research surround¬ 
ing best practices for professional learn¬ 
ing (Darling-Hammond & McLaughlin, 
1995; Darling-Hammond & Richardson, 
2009). However, due to the constraints 
of offering PD beyond our local region, 
an intensive summer workshop was the 
next best option to serve our out-of-region 
teachers. With all else being equivalent 
concerning the information that was pre¬ 
sented and the resources afforded to the 
participants, did the delivery format of the 
teacher professional development affect 
teacher and student program outcomes? To 
answer this question, data were collected 
on teacher learning, teacher implementa¬ 
tion, and student learning from each of 
the two different professional develop¬ 
ment formats based on Guskey’s model. 
The study presented here did not use an 
experimental design. The comparison was 
between the two PD models. In an effort 
to more clearly understand the relation¬ 
ship between the professional develop¬ 
ment and its impact on student learning, 
data from several sources were collected 
and analyzed. These included data related 
to teacher learning and satisfaction follow¬ 
ing the PD, what was implemented in the 
classroom and to what level, and, most 
importantly, measures of student learning. 
Each will be discussed below. 

Study Participants 

This study summarizes results from 
38 teachers who participated between 
the two POD PD programs. Teachers in 


the academic year (AY) program were 
from schools in one southwestern state, 
while teachers from summer institute 
(SI) program were from schools in the 
entire southwest region. Participant de¬ 
mographics varied to some degree across 
programs. While both PD formats in¬ 
cluded teachers at high school (HS) and 
middle school (MS) levels, the majority 
of the participants were high school sci¬ 
ence teachers. Teacher demographic data 
are shown in Table 1. 

From the original 38 teachers, a total of 
238 students from 24 teachers’ classrooms 
(16 in SI and 8 in AY) agreed to participate 
in this study focused on the connection 
between PD and student outcomes. 

Teacher Learning and Satisfaction 

At the conclusion of each PD session, 
teacher participants were asked to com¬ 
plete an online survey, an assessment 
covering content and spatial reasoning, 
and a GIS performance assessment. Par¬ 
ticipant satisfaction was measured with 
a survey which used a Likert scale with 
ratings 1-5 as poor, fair, average, very 
good, and outstanding. All participants 
from both AY and SI formats ranked 
the institute as very good, score of 4, to 
outstanding, score of 5. 

POD project staff developed a con¬ 
tent assessment to assess participants’ 
changes in understanding of science con¬ 
tent, scientific analysis, and spatial analy¬ 
sis. The specific categories (and number 
of questions) covered in this assessment 
were Earth science content (17), general 
science content (3), data analysis involv¬ 
ing charts and/or maps (5), making claims 


Table 1: Demographics of Teacher Participants by Percentage 



Summer Institute (SI) (n = 23) 

Academic Year (AY) (n=~\5) 

Teaching Level 

Middle School 

13% 

27% 

High School 

87% 

73% 

Content Area 

CTE 

30% 

27% 

Mathematics 

9% 

13% 

Science 

57% 

47% 

Social Studies 

4% 

7% 

Special Education 

0% 

7% 

Classroom Demographics 

Native American 

2% 

31% 

Latino/Hispanic 

35% 

6% 

Caucasian/Other 

63% 

63% 
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Table 2: Level of GIS Proficiency 


Level _ Definition _ 

Level 0 Inability to use the map or data to obtain information to answer the question. 

Level 1 Able to use the map and/or data to obtain information to answer the question. 

Level 2 Able to use the map and/or data to obtain information to answer the question and to 

create a basic map adding points, lines and polygons to the map to represent geographic 
features. 

Level 3 Able to use the map and/or data to obtain information to answer the question and create 

a basic map, add points, lines and polygons to the map to represent geographic features 
and symbolize geographic features based on levels of variability in data across a region 
(choropleth map). 

Level 4 Able to use the map and/or data to obtain information to answer the question and 

create a basic map, add points, lines and polygons to the map to represent geographic 
features, symbolize geographic features based on levels of variability in data across a 
region (choropleth map) and create a layout with a graphic (bar graph or pie chart) and/or 
include other graphical representations to communicate ideas. 


based on evidence from maps produced in 
ArcGIS software (5), and spatial analysis 
(10). Pretest data were gathered prior to 
the implementation of the program and 
post-test data were gathered at the last 
meeting of each institute. Mean content 
scores improved for almost all partici¬ 
pants across both programs. A Mann- 
Whitney U test was conducted to evaluate 
whether there was a difference in teach¬ 
ers’ content understanding between the 
two programs, as measured by a post pro¬ 
gram content assessment. The results of 
the test were not significant, z = -1.68, 
p = 0.10; the teachers in both the AY and 
SI PD learned an equivalent amount of 
content based on the post-test. 

Figure 1 


A geospatial performance assess¬ 
ment was administered pre- and post¬ 
institute to teacher participants. This 
assessment measured participants’ 
abilities to use the ArcGIS software. 
Ability levels were defined from 0 
through 4 in terms of proficiency as 
shown in Table 2. 

Participants in both institutes showed 
growth in GIS skills from pre- to post¬ 
institute (Figure 1). Results for the SI 
participants were higher than those in 
the AY institute. All participants took the 
GIS performance assessment on the last 
day of their respective institute rather 
than at the same time in the academic 
year of implementation. 


Overall results for both groups are 
shown in Table 3. The groups are very 
similar showing high satisfaction and 
improved learning following the pro¬ 
fessional development. Without larger 
numbers of participants in each group to 
compare, we have general trends but are 
reluctant to make any stronger claims. 

Organizational Support 

In order to ensure support for teachers 
during implementation, recruitment ef¬ 
forts focused on pairs of teachers at the 
same school site. However, when partici¬ 
pants returned to their respective schools, 
IT and administrator support varied. Data 
on technology support were collected in¬ 
formally through inquiries, emails, phone 
calls, discussion board postings, and for¬ 
mally through surveys. Some schools 
had more support and access than others 
before they even began implementation, 
but the problems installing software were 
similar across the two formats of PD. 
Regardless, all teachers had the software 
installed and ready to use within four 
months of the PD sessions (Table 4). 

Implementation 

Levels of implementation were mea¬ 
sured using multiple data sources and 
multiple researchers to triangulate pat¬ 
terns and themes (Creswell & Miller, 
2000). Data sources included surveys, 
classroom observations, and collection 
of classroom artifacts to describe both 
the quantity and quality of implemen¬ 
tation following the POD PD institutes. 
Because teachers must implement les¬ 
sons in order for student achievement 
to improve, data were collected on the 
level of classroom implementation. All 
teachers in the study who completed the 
AY program implemented at least one 
lesson, while only 81% of the SI partici¬ 
pants implemented a lesson. 

For those 24 teachers who imple¬ 
mented at least one lesson, results of im¬ 
plementation were further examined to 
determine which teachers implemented 
which types of lessons - a lesson out 
of a book with data provided, a general 
project designed by POD program staff 
with data provided, or a project with stu¬ 
dents collecting their own data. Results 


100 % 



■ Academic Year 
Summer 


Level 0 Level 1 Level 2 Level 3 
Level of Performance 


Level 4 


Figure 1: Comparing level of geospatial performance between programs at the end of professional 
development. 
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Table 3: Means on Post PD Teacher Questionnaire and Assessments 



Summer Institute (SI) (n = 16) 

Academic Year (AY) (n = 8) 

Satisfaction with Institute 3 

4.6 

4.5 

Content Assessment 6 

32.4 

29.9 

GIS Performance Assessment 3 

3.5 

2.8 


a Based on a 5-Point Scale (1-5) 
b Based on a 40-Point Scale (1 -40) 
c Based on a 4-Point Scale (1 -4) 


are reported based on the “highest” level 
of project implemented by the teacher 
in their classroom in Table 5. For those 
teachers who implemented lessons, the 
lessons were similar in nature when 
compared across the two PD formats. 

To attempt to quantify the quality of 
implementation, classroom observa¬ 
tions were conducted using a modified 
instrument based on Inside the Class¬ 
room Observation and Analytic Protocol 
(Horizon Research, Inc., 2000). To en¬ 
able comparisons across projects, com¬ 
mon sections of implementation from 
the protocol were chosen as a focus 
(Table 6). Observers were looking for 
evidence of high quality teaching, based 
on the degree of student-centered teach¬ 
ing as opposed to direct instruction, and 
the degree to which inquiry was valued 
and encouraged. 

Comparing the means of the obser¬ 
vation scores on the six common items 
(Table 6), the scores are very consistent. 


The average score of the teachers in the 
SI was 9.1 and the average score of the 
teachers in the AY was 10.1. 

Student Understanding 

The measure of effective teaching is im¬ 
proved student understanding. Measures 
of student understanding from this study 
included pre- and post- tests and student 
artifacts gathered from the teachers fol¬ 
lowing implementation of the units. In 
both projects, students increased their con¬ 
tent understanding over the course of the 
project, as measured by the 40-question 
pre-post content assessment. This was the 
same test administered to their teachers. 

For students whose teachers had par¬ 
ticipated in the SI project, there was a 
significant increase in achievement from 
pre-test to post-test, f(145) = -5.37, p < 
.001, Cohen’s d = .44. In the AY proj¬ 
ect, there was also a significant increase 
in achievement from pre-test to post¬ 
test, r(93) = -5.59, p < .001, Cohen’s 


Table 4: Organizational Support 


Summer Institute (SI) (n=23) 

Academic Year (AY) (n=15) 


Participants 

23 teachers 

15 teachers 

12 schools represented 

8 schools represented 

11 teams 

7 teams 


Software Installation 

June or July PD 

September PD 

October 2010 (4 months following PD) 

January 2011 (4 months following PD) 

10 schools installed 

7 schools installed 

1 school not installed 

1 school never installed or perceived as 
no access only using Macs and ArcView 3x 

1 school dropped out 

Partnerships and Level of Participation 

10 teams both partners fully engaged 

4 teams both partners fully engaged for entire 

during institute 

AY institute 

1 single teacher with no partner 

1 single teacher with no partner 

1 teacher lost partner, due to illness 

1 teacher lost partner towards the end 

1 team - partner disengaged 

1 team dropped out halfway through the year 

2 teams - partner disengaged 

1 team partner had health issues and missed 
several classes, but finished program 


d =.56. Nonetheless, there was no sig¬ 
nificant difference in student content 
understanding regardless of whether 
their teachers participated in AY or SI 
institutes as measured by a content post¬ 
assessment, ?(238) = .614, p = .540. In 
other words, there was not a significant 
difference in student achievement be¬ 
tween the AY and the SI programs. 

Student understanding was also evalu¬ 
ated in terms of student work samples 
submitted. Teachers were instructed to 
implement two project-based learning 
modules, which required students to use 
GIS to understand a scientific concept 
through examination of data and to use 
GIS to visually represent claims based on 
data. A variety of student work samples 
were submitted, including worksheets, 
paper and pencil assignments, paper as¬ 
sessment maps, projects, fieldwork, au¬ 
thentic projects, and analysis reports. Note 
that although 238 students from 24 teach¬ 
ers completed the pre- and post-tests, only 
161 student work samples from 9 teachers 
in the SI model and 8 teachers in the AY 
model were submitted for further analysis. 

POD project staff developed a rubric 
to score student work samples on use 
of data or GIS, content application, and 
claims and evidence (Table 7). 

Student work demonstrated that ArcGIS 
was utilized to varying degrees and the 
quality of student work products varied. 
Student artifacts were categorized as ei¬ 
ther resulting from implementation of: 1) 
a basic lesson - usually a worksheet pro¬ 
vided in PD; 2) a general project that POD 
project staff designed for the unit; or 3) an 
authentic project where students gathered 
data and attempted to solve a real-world 
problem. Project scores by type of work 
submitted are listed in Table 8. 

In general, any type of student project 
yielded higher overall scores in all areas 
compared to basic lessons. However, 
more teachers implemented authentic, 
relevant projects in the AY program 
(63%) than in the SI program (44%). 

Findings: Summer Institute (SI) 
vs. Academic Year (AY) 

Implementation is integral for PD to 
lead to student learning outcomes. The 
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Table 5: Highest Level of Project or Lesson Implementation in the Classroom by Teacher Partners 



Summer Institute (SI) (n = 16) 

Academic Year (AY) (n = 8) 

No implementation 

3(19%) 

N/A 

Basic lesson 

2(13%) 

2 (25%) 

General project 

4 (25%) 

1 (13%) 

Authentic project 

7 (44%) 

5 (63%) 


focus of this research was to see how the 
format of the professional development, 
AY vs. SI, affected student learning. With 
the focus on student learning, the types and 
quality of student products were measured 
as evidence of the effectiveness of the PD. 
Indicators showed that both PD formats 
yielded similar learning outcomes for both 
teachers and their students. 

Teacher Outcomes 

1. There were high levels of teacher 
satisfaction with PD in both 
formats. 

2. Teachers in both formats 
improved GIS skills and content 
knowledge at a similar level. 

3. A number of teachers in both for¬ 
mats were able to implement les¬ 
sons using GIS that demonstrated 
exemplary teaching practices. 

Student Outcomes 

1. Students significantly improved 
content knowledge as measured 
by an achievement test in both 
formats, with no difference in 
achievement between the formats. 

2. Student work demonstrated 
ArcGIS was utilized to varying 
degrees and the quality of student 
work products varied, but the 
means for the student work sam¬ 
ples did not vary significantly. 


3. Lessons with authentic problems 
promoted a higher level of stu¬ 
dent engagement. 

Overall, there were few observed dif¬ 
ferences between the teachers who par¬ 
ticipated in the two PD formats or the 
overall measures of student learning. 

Insights on Implementation 

Teachers and students showed im¬ 
provement in all areas assessed, but there 
was not an observable effect on student 
understanding based on the delivery for¬ 
mat of the PD. However, the PD format 
may affect levels of implementation, 
which is important if the assumption is 
that changes in student understanding 
are only observed after teachers imple¬ 
ment interventions learned during PD. 
One difference between the formats 
was that classroom implementation was 
higher in the AY program. In the SI pro¬ 
gram, using initial participant numbers, 
43% did not complete all requirements 
(attend all PD sessions and implement 
at least one lesson), compared to 20% in 
the AY program who did not complete 
requirements. A higher number of AY 
participants implemented lessons in their 
classrooms following PD. It seems that 
the structure of the PD model had little 
impact on student or teacher learning, but 
follow-up meetings in the AY group did 
provide teachers support and incentive 


to complete the program and therefore 
implement lessons with students. 

Additionally, teachers in both PD for¬ 
mats were encouraged to not only imple¬ 
ment technology-integrated lessons with 
data provided, but also to implement a 
project-based lesson, and then to facilitate a 
site-based, authentic project surrounding 
their individual schools and communities. 
In this study, the rate of implementation of 
any lessons during or following PD was 
higher with the AY program, with 100% 
(8/8) implementation rate compared to 
81% (13/16) implementation rate for the 
SI program. The lower implementation 
rate in the SI program was surprising, be¬ 
cause the SI teachers had the opportunity 
to plan the implementation before their 
school year started, presumably helping 
to better integrate the lessons into their 
regular teaching agendas. 

Additionally, there appeared to be a 
slightly higher average application of 
science content in the student projects 
submitted by AY participants. Imple¬ 
mentation of authentic projects did not 
necessarily yield higher GIS, content, 
and inquiry skills, but if a structured les¬ 
son was provided preceding an authentic 
one, an improvement was observed in 
the second product. Based on anecdotal 
evidence from project participants, stu¬ 
dents who investigated authentic prob¬ 
lems and collected their own data were 
more engaged and involved in project 
activities than those who were solving 
general problems or completing lessons 
out of a text, which may explain the dif¬ 
ferences in student content scores on 
their projects. Here are responses from a 
teacher regarding her submission of stu¬ 
dent work for analysis: 

“As can be seen from the Power 
Points submitted to you, there were 
quite a few projects that did not in¬ 
clude GIS maps at all. Generally, 
they do not care.” - Teacher in SI 
program, following general project 
implementation 

“The students were very interested 
in this project because it was a real 
world application of classroom infor¬ 
mation. They saw that their research 


Table 6: Domains and Items Common to Both POD Project Observation Protocols 


Implementation _ 

The instructional strategies were consistent with investigative mathematics/science. 

The teacher appeared confident in his/her ability to teach mathematics/science. 

The teacher’s questioning strategies were likely to enhance the development of student conceptual 
understanding/problem solving (e.g., emphasized higher order questions, appropriately used “wait time,” 
identified prior conceptions and misconceptions). 

Mathematics/Science Content _ 

Students were intellectually engaged with important ideas relevant to the focus of the lesson. 

Appropriate connections were made to other areas of mathematics/science, to other disciplines, and/or to 
real-world contexts. 

Classroom Culture _ 

The climate of the lesson encouraged students to generate ideas, questions, conjectures, and/or propositions. 
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Table 7: Rubric for Scoring Student Work Samples 


Level 

Access/Use of data/ArcGIS 

Content 

Claims and Evidence 

1 

Get and display information 
from any source. 

Specific facts collected/ 
presented but unrelated 
to science concept. 

Can’t tell what content 
was to be learned. 

Claim without any 
supporting data as 
evidence or explanation. 

2 

Get and display information 
from ArcGIS. 

Science concept mentioned 
but not applied. 

Claim with explanation but 
no supporting data/ 
evidence. 

3 

ViewArcGIS data (build layout, 
layers, table). 

Science term/fact applied. 

Claim with any supporting 
data as evidence 

4 

Use evidence map/visual to 
illustrate a claim. Make sense 
of data, show patterns, clearly 
present information. 

Science concept applied 
to the situation and 
discussed more generally. 

Discusses more than 1 piece 
of evidence to support 
claim. 

5 

Create evidence map/visual 
representation of data from 
scratch to illustrate claim. 

N/A 

Claim with own or class 
collected data/experiments, 
more than one listed as 
evidence to support this 
claim and evidence was 
used to make a decision. 


and ideas for repair and prevention 
could have a positive impact on their 
school. In addition, they were equally 
excited about using the LabQuests. 
This was an opportunity for them to 
collect real data in the field.” - Same 
teacher in SI program following site 
based authentic assignment 

Teachers in the SI PD were required to 
have students create presentations that 
included data as evidence for claims 
about the most preferred time to visit a 
city, based on weather and climate pres¬ 
ent at different times of the year. The 
differences observed between SI student 
projects seemed to depend on how the 
lessons were presented to students, what 
the teachers expected from the student 
work, and what the teachers were empha¬ 
sizing in the grading or evaluation of the 
work. Since the AY PD was implemented 
following the SI, assessment and creation 
of rubrics that emphasized claims and evi¬ 
dence and application of science content 
was incorporated into the AY PD. The AY 


teachers were provided time and instruc¬ 
tion on assessing student learning that in¬ 
cluded examining student work to ensure 
that the project required a focus on science 
content over geography, which seemed 
like an important modification in the PD 
to support classroom implementation. 

Generally, teachers and students showed 
improvement in all areas assessed, but there 
was not a measurable effect on student 
understanding until implementation was 
more closely examined. The AY format 
supported higher levels of implementation 
following PD with higher fidelity. It seems 
that examining implementation levels in 
conjunction with the student work sam¬ 
ples submitted provided better insights as 
to how to support teachers to implement 
project-based learning modules using geo¬ 
spatial technologies. 

Summary 

Lessons learned from our study con¬ 
tribute to the growing body of evidence 
that geospatial technology supports 


student learning and provides an oppor¬ 
tunity for students to learn through ex¬ 
amining authentic problems. However, 
even when teachers were satisfied with 
the PD and were provided technology 
support, many factors affected the level 
of implementation in the classroom. 
Our findings suggest that the struc¬ 
ture of the PD model had little impact 
on student or teacher learning, but 
follow-up meetings in the AY group 
did provide teachers support and 
incentive to implement lessons with 
students at a higher rate than the SI 
group. 

Several other mitigating factors may 
have contributed to the increased imple¬ 
mentation rates seen in the AY group. 
Teachers in the AY program had monthly, 
face-to-face, follow-up support as they 
implemented their lessons. This appears 
to have been a critical factor for teachers 
as they taught new lessons integrating geo¬ 
spatial technology. Additionally, the AY 
participants were asked to present what 
they had implemented at their final PD 
meeting and possibly felt more self- 
imposed pressure to actually implement 
the lessons, which is a crucial link to ex¬ 
pecting change in classrooms (Guskey, 
2002). There were also monetary incen¬ 
tives tied to attendance and implementa¬ 
tion of the lessons in the AY group, which 
were not present with the SI group. 

Loucks-Horsley et al. (2003) assert that 
on-going support is vital for science and 
math PD, and this may be especially criti¬ 
cal for teachers utilizing new technology 
in teaching (Trautmann & MaKinster, 
2009). Creating teacher accountability 
measures beyond the PD institute was not 
obvious to us until this study. Therefore, 
a hybrid model of PD, with incentives, 
that includes both face time and year- 
round support, may be the best model to 
encourage high levels of implementation 
as participants learn to integrate new tech¬ 
nology into their teaching. 

Cheng and Hanuscin’s Taxonomy of 
Hybrid Professional Development Model 
(2012) outlines a continuum of PD models 
in which technology is utilized to support 
teacher learning, with technology integra¬ 
tion levels ranging from "high/interactive” 
to “low/information based”. In the high/ 


Table 8: Average Student Rubric Scores in Each Area by Assignment Type 



Summer Institute (SI) (n ■■ 

= 118) 


Academic Year (AY) (n ■■ 

= 43) 

n 

GIS 

Content 

Claims 

n 

GIS 

Content 

Claims 

Basic lesson 

2 

1 

2 

1 

22 

2 

2 

1 

General Project 

41 

2 

2 

3 

9 

3 

3 

2 

Authentic Project 

75 

4 

1 

2 

12 

2 

3 

3 
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interactive model, online interaction is 
essential to the program. In the low/infor¬ 
mation model, the main delivery method 
is face-to-face; online technology plays a 
minor role. Further recommendations for 
increasing the effectiveness of hybrid PD 
from the research community include: of¬ 
fering some asynchronous activities to en¬ 
able a more learner-centered, personalized 
experience for teachers (Fritschi & Wolf, 
2012); building a shared vision to promote 
ownership of the project goals and to en¬ 
courage structured collaboration on com¬ 
mon projects (Fritschi & Wolf, 2012; Ocker 
& Yaverbaum, 1999; Owston, Wideman, 
Murphy, & Lupshenyuk, 2008; Vrasidas 
& Zembylas, 2004); providing mentors 
and regular feedback on authentic, job- 
embedded tasks (Boling, Florida, & Martin, 
2005; Johnson, 2001; Ocker & Yaverbaum, 
1999; Owston et al, 2008; Vrasidas & 
Zembylas, 2004); facilitating metacogni- 
tive activities for participants to reflect on 
the experience (Owston et al., 2008; Vrasidas 
& Zembylas 2004); and providing some 
face-to-face time together before work¬ 
ing collaboratively online (Johnson, 2001; 
Ocker & Yaverbaum, 1999; Owston et al., 
2008). Based on these recommendations, a 
hybrid format for PD should include some 
face-to-face interactions to initiate partici¬ 
pants into the project and allow for rela¬ 
tionship-building that will continue online. 

The online portion of the PD should 
be closer to the “high” end of Cheng and 
Hanuscin’s proposed spectrum, with a 
mixture of asynchronous activities that 
meet participant needs, and synchronous 
interactions in the form of webinars or 
break-out rooms where the technology 
becomes “invisible” or at least second¬ 
ary to the professional learning that 
occurs, to allow a more face-to-face 
authenticity to the online environment 
(Fritschi & Wolf, 2012; Owston et al., 
2008; Vrasidas & Zembylas 2004). The 
tools within which the online environ¬ 
ment is conducted must be reliable, and 
teachers must be supported while learn¬ 
ing and using the new collaboration tools 
(Boling, Florida, & Martin, 2005; Ocker 
& Yaverbaum, 1999). With this in mind, 
the delivery format may actually mat¬ 
ter if teachers’ participation in PD can 
develop knowledge and skills, and then 


be followed by the use of technology to 
enhance year-long interactions and ac¬ 
countability to the project. 

Teachers must implement knowledge 
and skills learned during PD in their 
classrooms for student learning to oc¬ 
cur. Professional development providers 
must support and encourage implemen¬ 
tation in classrooms. One way to do that 
might be to make teachers accountable 
to the project, and therefore feel obligated 
to implement some of what was learned 
in PD in their classrooms. If face-to- 
face meetings are not possible, a hybrid 
model might help accomplish this goal. 

Even when teachers apply knowledge 
and skills learned during PD in the school 
year immediately following the PD experi¬ 
ence, there is no guarantee they will con¬ 
tinue to implement once support from the 
PD project ends. Although the POD PD 
projects have ended, we are currently con¬ 
ducting additional research to determine 
what instructional practices from PD are 
persisting one to two years following par¬ 
ticipation in PD. In addition, we are work¬ 
ing to describe contexts that encourage, 
support, and maintain the use of geospatial 
technologies and project based instruction 
when the PD institutes and participant sup¬ 
port end. 
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